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intervention activity directed at problem behaviors in a representative 
sample of the nation's schools. Five kinds of information were collected: (1) 

examples were collected of prevention and intervention models being used; (2) 
principals identified activities their schools had in place to prevent 
delinquency, drug use, and other problem behaviors; (3) individuals 
knowledgeable about school prevention activities were surveyed to obtain 
detailed descriptions of specific prevention activities; (4) teachers and 
students were surveyed to obtain reports of their own participation in 
prevention activities; and (5) principals were surveyed a second time about 
disciplinary actions and crime in their schools. Activity questionnaires were 
used to obtain detailed descriptions of the nature, level, and quality of 
implementation of specific prevention activities. From the total sample of 
more than 17,000 prevention activities identified, one activity in each of 14 
categories was sampled. Questionnaires were sent for about 7,100 activities 
and 3,700 were completed. All D.A.R.E. and peer mediation programs were 
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Design and Implementation of a Large-Sample Study of Schools 

I will describe the sample of schools in the National Study of Delinquency 
Prevention in Schools. I will also describe (a) questionnaire surveys developed for 
principals, teachers, students, and program implementers; (b) who provided information 
and (c) how schools were recruited to participate and how many participated. 

The study was undertaken to develop a comprehensive account of the levels of 
problem behavior in U.S. schools and of what schools do to prevent problem behavior 
and to promote a safe environment. The study contrasts with much evaluation 
research that is directed at assessing the effectiveness of specific practices. Our 
research was not undertaken to assess the effectiveness of specific instances of 
prevention or intervention activities. Much evaluation research examines isolated 
programs and seeks to determine their effects. But many program evaluations lack 
ecological validity. Because typical evaluations are designed to focus on one or a small 
number of specific realizations of a program or practice, they lack a sufficiently 
representative design to describe typical practices or the typical degree of strength and 
integrity attained when programs are applied outside of the experimental context. In 
contrast, the present research was designed to assess the nature, extent, and quality of 
prevention and intervention activity directed at problem behavior and school safety in a 
representative sample of the nation’s schools. 

Research has shown that some kinds of interventions to reduce problem behavior 
can be effective. For example, behavioral and cognitive-behavioral interventions have 
repeatedly been shown to be effective in reducing problem behavior or improving 
attendance. In many cases, however, the evidence derives from optimal or at least 
good implementations of the intervention in question. Often investigators train 
implementers, monitor their behavior, correct implementation errors, or are directly 
involved in the application of the method being studied. In some cases, the evidence is 
derived from schools that were especially amenable to program implementation. 

In research programs, the capacity of the school to serve as an implementation site 
is likely to be greater than the typical school - evidenced at least in part by its 
willingness to participate in a research project. In addition, the particular implementers 
(teachers or others) are likely to be selected for their willingness to implement a 
program and cooperate with evaluators, and quick ability to learn to put new methods in 
place. In all of these respects, they are likely to produce better instances of 
implementation than would be achieved in the average school, let alone schools where 
many youths are engaged in high levels of problem behavior or where faculty are 
demoralized. 

Schools and their personnel differ in the extent to which they are able or willing to 
produce strong and faithful implementations of intended programs. For example, in a 
study of eight urban schools, Botvin and his colleagues reported that the amount of Life 
Skills Training material covered by teachers ranged from 44% to 83%. Positive effects 
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of the program were found only for a high implementation group, not for the low 
implementation group. In another study by Botvin and colleagues, coverage of the 
curriculum ranged from 27% to 97%, with 75% of students exposed to 60% or more of 
the material. The level of implementation was related to the effectiveness of the 
intervention. 

Health and mental health researchers refer to the distinction between intervention 
efficacy (an efficacious intervention can work) and effectiveness (how well the 
intervention does work when applied in typical settings by typical practitioners). In this 
language, some interventions to reduce or prevent problem behavior have been shown 
to have efficacy, but almost no interventions have been shown to be generally effective. 
If efficacious interventions are ineffective, it is likely that flawed implementation is a 
large part of the reason. Our project focuses on implementation. 

Hypotheses 

With this perspective on the importance of the problem of implementation in 
representative settings, a number of factors were hypothesized on the basis of prior 
research or experience to be linked to the successful implementation of prevention 
programs. These are summarized in the first figure. 

1 . Organizational capacity (morale, staff stability, history of failed or successful 
programs in the past). Better morale, more stable staff, and a history of successful 
program implementation is expected to go with better current implementation. In 
contrast, low morale, high staff or principal turnover, and a history of failed 
programs is expected to go with poor implementation. 

2. Leadership and staff traits and past accomplishments. Implementation is expected 
to be better in schools in which principals report that they display behaviors 
associated with effective leadership and where they are perceived by others as 
effective leaders. Schools where principals or program implementers have a 
record of accomplishment in the past are expected to be more successful in what 
they currently implement. And programs implemented by more conscientious 
implementers in schools led by more conscientious principals are expected to be 
better implemented. 

3. Budget and resources. Lack of adequate budget or resources is expected to thwart 
successful program implementation. 

4. Organizational support. Extensive and high quality training is expected to promote 
high quality and extensive implementation, whereas lack of training or poor training 
is expected to lead to weak or poor quality implementation. Direct and extensive 
supervision is expected to lead to higher quality and more complete 
implementation. Principal support for an activity is expected to lead to more 
extensive implementation and to higher quality implementation. 
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5. Program structure, manuals, implementation standards, and quality control 
mechanisms. Greater structure is expected to lead to higher quality 
implementation and implementation that more closely follows a plan for what 
should be implemented. Implementation manuals can provide scaffolding for 
implementers by providing structure, an organization, and a plan for what to do as 
well as guidance on how to do it. Prepared materials, such as handouts, overhead 
masters, and videotapes, can make implementation easier and deviation from 
intended content less likely. Statements of standards for implementation provide 
the persons implementing a program with a basis for determining whether what is 
being done is good enough. And quality control mechanisms such as procedures 
for monitoring progress, review of progress, and worker supervision are expected to 
promote better implementation by focusing attention on how well implementation is 
being done. 

6. Integration into normal school operations, local initiation, and local planning. The 
extent to which program design choices are integrated with normal school 
operations is expected to have consequences for implementation. Better 
integration of activities with the regularities of the school is expected to lead to 
more enthusiastic and widespread adoption of prevention practices within a school. 
Schools employ teachers, supervised by principals, to carry out instruction. 

Schools also sometimes utilize the services of volunteers or other persons not 
employed by the school. The school has much less control over the timing, 
duration, and extent of involvement of external personnel than it does over regular 
employees. When prevention activity is carried out by regular school employees in 
the conduct of their routine work, it is more likely to be widely implemented. 

When locally planned or initiated, activities are (by definition) not imposed upon a 
school. Accordingly, impulses to resist adoption or implementation — which are 
sometimes triggered by programs imposed upon a school — are less likely to be 
evoked by locally planned programs. 

When school personnel use information derived from researchers, experts, 
publications, and other sources about how to implement activities, they are 
expected to incorporate more best practices and to emulate successful models 
more fully because they are more likely to have the information needed to do so. 

7. Program feasibility (match between program design features and regular activities 
of schools, few obstacles). Some activities or arrangements are expected to 
encounter obstacles to implementation. Activities that occur after the end (or 
before the beginning) of the regular school day or on weekends will be more 
difficult to implement because they are outside of regular work hours, for example. 

8. Level of disorder. We hypothesized that high levels of disorder in a school will 
make everything more difficult to implement. 



3 




5 



Plan of the Research 



The plan for the research called for the collection of five main kinds of information 
by executing an equal number of steps. The second figure shows a summary. 

First, examples of prevention and intervention models being used in schools were 
collected, examined and classified to develop a comprehensive taxonomy of activities. 
This first step was completed at the end of 1996. Figure 3 is adapted from the 
complete taxonomy. Plans were made to obtain detailed information about the first 14 
categories shown in this figure. 

Second, principals in a national probability sample of schools were surveyed to 
identify activities their schools had in place to prevent or reduce delinquency, drug use, 
or other problem behavior or to promote a safe and orderly school environment. They 
indicated if their school had activities of various types, named the activities, and 
provided the names of individuals who could provide details about each activity named. 
The resulting lists of school prevention and intervention programs were used to sample 
prevention activities in a subsequent step. Principals also described features of their 
schools and reported on past experiences with the implementation of programs and on 
school staffing. These surveys were conducted in the spring, summer, and early fall of 
1997. 

Third, individuals knowledgeable about school prevention activities (called “activity 
coordinators”) were surveyed to obtain detailed descriptions of specific prevention 
activities and to describe certain features of their school. To conduct these surveys, we 
developed a set of fourteen activity coordinator questionnaires corresponding to 
fourteen categories in our taxonomy of prevention models. To the extent possible, the 
questionnaires for all categories were parallel. Thus, although the specific content of 
questionnaires for different areas was appropriate for activities of each type, the nature 
of information sought was parallel. Wherever possible each questionnaire sought 
information about the extent to which best practices were used, about the 
extensiveness of student exposure, about training, and so forth. Activity coordinators 
also reported about themselves and about school support and supervision for 
prevention activities. These surveys were conducted in the spring of 1998. 

Fourth, teachers and students in participating secondary schools were surveyed to 
obtain their reports of their own participation in prevention activities, about prevention 
activities in the school, and to obtain reports about victimization, safety, delinquent 
behavior, school orderliness, and other aspects of school climate. These surveys were 
conducted in the spring of 1998. Generally, all teachers in participating schools were 
sampled, and a sufficient number of students were sampled to produce an estimated 
50 respondents per school. 

Fifth, principals were surveyed for a second time in the spring of 1998. They 
reported about school-wide disciplinary policies and practices, crimes occurring in the 
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school, certain school-wide arrangements such as scheduling, architectural features of 
the school, and other characteristics of the school about which the principal was the 
most appropriate informant. Principals also reported about their own practices, 
biographical history, and personality style. 

Figure 4 summarizes the surveys conducted and the type of information collected in 
each. Not shown in the figure, we also used certain archival information - drawn from 
the Common Core of Data maintained by the U.S. Department of Education, provided 
by the mailing list vendor, or obtained from the 1990 census of population. 

The sample was designed to describe schools in the United States and to describe 
schools by level and location. Accordingly a sample of public, private, and Catholic 
schools, stratified by location (urban, suburban, and rural) and level (elementary, 
middle, and high) was drawn. A probability sample of 1287 schools was selected with 
the expectation that if a response rate of 70% could be achieved there would be 300 
schools responding at each level and 300 schools responding from each location. 

Conducting Surveys and Participation Rates 

In conducting the phase 1 principal survey, we determined that of the 1287 entities 
sampled, 7 were closed and one was not a school - leaving 1279 schools in the 
sample. Overall, useful responses were received from 848 schools in phase 1 principal 
survey, 66% of those from which responses were sought. Participation rates ranged 
from a low of 59% among urban high schools to a high of 75% among rural elementary 
schools. Participation rates and number of participating schools for each component of 
the project are shown in Figure 5. Participation at the level of student surveys was the 
most difficult to obtain - as was participation of urban high schools in general. 

The effort that was required to obtain completed questionnaires from schools far 
exceeded our expectations. Information about effort required to obtain data in phase 1 
is summarized in Figure 6. One indication of the difficulty involved are the counts of 
telephone contacts with schools that were required to obtain cooperation. In all, we 
completed 8,783 telephone calls to schools to request phase 1 principal data. The 
number of calls per school ranged from 0 (some schools returned questionnaires 
without having to be called) to 36. The average number of telephone calls made to 
schools that had to be called at least once was 7.9 completed calls. In addition, survey 
materials were resent once by Federal Express to 964 schools that had not responded. 

Many schools still indicated to our callers that they had not received or had 
misplaced the questionnaires, and our response was to mail another set of 
replacements. This was done for 531 schools (42% of the sample). Replacements for 
“lost" questionnaires were resent twice to 1 18 schools (9%) and three times to 21 
schools (2%). When we could obtain school telefax numbers, we sent faxes to 
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nonresponding schools. One telefaxed request was sent to 225 schools and two 
telefaxed requests were sent to 13 schools. 

In view of the difficulty in obtaining data from schools, we sought ways to bring 
greater resources to the research. NIJ personnel assisted us in working with the 
Department of Education to bring about a merger of our ongoing study and resources 
intended to address similar problems in the form of a contract ED had with Westat to 
gather information about school violence and programs sponsored by the Safe and 
Drug Free Schools and Communities Act. 

In conducting the phase 2 surveys, an additional school was found to have been 
closed, leaving 1278 schools in the sample. Again, obtaining cooperation was most 
difficult in urban schools, where completed phase 2 principal questionnaires were 
obtained for 46% of the sample. Rural schools were more cooperative, and we 
obtained completed phase 2 principal questionnaires from 57% of rural schools. 
Participation ranged from a low of 40% for urban high schools to 58% for rural middle 
grades schools. Figure 5 (again) summarizes participation rates. 

We sought the completion of student questionnaires in all secondary schools. 
Usable questionnaires were completed by over 16,000 students. Schools with poor 
levels of student participation are treated as nonparticipants in the figure. Overall, 36% 
of the secondary schools from whom participation was sought in student surveys 
participated at a useful level. Participation ranged from a low of 23% of urban high 
schools to 50% of rural middle schools. 

We sought the completion of teacher questionnaires in all secondary schools, and 
usable questionnaires were completed by over 13,000 teachers. Rural schools were 
much more cooperative than suburban or urban schools. Participation ranged from a 
low of 39% of urban high schools to 59% of rural middle schools. 

Activity questionnaires were used to obtain detailed descriptions of the nature, 
level, and quality of implementation of specific prevention activities. From the total 
sample of more than 17,000 prevention activities identified in phase 1, we sampled one 
activity in each of 14 categories per school. In addition, we sampled all D.A.R.E. and 
peer mediation programs because of special interest in these particularly popular 
prevention programs. We sent questionnaires for about 7, 1 00 activities. Of these, 
almost 3,700 were completed (46% of all sampled activities and 52% of the activities for 
which information was requested). 



Reference 
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Hantman, I. (2000). National study of delinquency prevention in schools (Final 
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Hypothesized Factors Leading to Successful 
Program Implementation 

The following foster successful implementation of prevention 

programs: 

★ Organizational capacity (morale, history of few failed 
programs in the past, staffing stability) 

★ Leadership and staff traits, past accomplishments 

★ Budget and support 

★ Organizational support (training, supervision, principal 
support) 

★ Program structure — manuals, implementation 
standards, quality control mechanisms 

★ Integration with normal school operations, local 
initiation, local planning, local information use 

★ Program feasibility — match between program design 
features and regular activities of the implementing 
school 

★ Little disorder 

“For whoever has , to him more will be given , and he 
will have abundance; but whoever does not have , even 
what he has will be taken away from him Matthew 
13:12 
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Project Design 



Development of taxonomy of prevention activities 
Collected, examined, and classified examples of 
prevention models being now being used in schools 

Principal survey for program identification 
(spring and summer 1997) 
identified program implementers 
described features of schools 

reported on experiences with implementation & staffing 

Activity coordinator survey 
(spring 1998) 

described programs and certain features of the school 

Teacher and student surveys 
(spring 1998) 

described participation in prevention activities, school 
climate 

reported on victimization, safety, other features of the 
school 

Principal discipline and school-wide practices survey 
(spring 1998) 

described certain school-wide arrangements - especially 
discipline - and their own practices and style 
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A Classification of Prevention Activity 

1 . Prevention curriculum, instruction, or training 

2. Behavioral or behavior modification interventions 

3. Counseling/social work/psychological/therapeutic 
interventions 

4. Individual attention/mentoring/tutoring/coaching 

5. Recreational, enrichment and leisure activities 

6. Referral to other agencies or for other services 

7. Improved instructional methods or practices 

8. Improved classroom management methods or practices 

9. Distinctive culture or climate for interpersonal exchanges - 
or improvements to intergroup relations or interaction 
between school & community 

10. Use of external personnel resources in classrooms 

1 1 . Youth roles in regulating and responding to student conduct 

12. School planning structure or process - or management of 
change 

13. Security and surveillance 

14. Services to families 

15. Rules, policies, regulations, laws, or enforcement 

16. Provision of information 

17. Reorganization of grades, classes, or school schedules 

18. Exclusion of weapons or contraband 

19. Alter school composition 

20. Training or staff development intervention 

2 1 . Architectural features of the school 

22. Treatment or prevention interventions for administration, 
faculty, or staff 
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Measures Employed and Sources of Information 


What is measured 


Principal 


Teachers 


Students 


Iinple- 

menters 


Grade levels/Demographics 


■ 


■ 


■ 




School safety 




■ 


■ 




Victimization 




■ 


■ 




Drug use, violence, other 
delinquent behavior 


■ 




■ 




School climate — morale, 
administrator leadership, 
discipline related 


■ 


■ 


■ 


■ 


Level of implementation of 
activities 


■ 


■ 


■ 


■ 


Correlates of problem 
behavior 






■ 




Leadership style of 
principal 


■ 


■ 






Personality 


■ 






■ 


Biographical data 


■ 






■ 


Organizational origins of 
activities 


■ 






■ 


Funding sources 


■ 






■ 


Nature & extent of training 


■ 






■ 


Program features 


■ 






■ 


Staff stability vs. turnover 


■ 






■ 


Organizational capacity 


■ 


■ 




■ 
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Participation Rates and Numbers of Participating Schools 





% schools 


N schools 




participa- 


participa- 


Study component 


ting 


ting 


Principal survey, 1997 


66 


848 


Principal survey, 1998 


50 


635 


Student survey, 1998 


37 


310 


Teacher survey, 1998 


48 


403 


Activity coordinator survey, 1998 


43 


554 




% 


N 




activities 


activities 




participa- 


participa- 




ting 


ting 


Activity coordinator survey, 1998 


52 


3691 



Note, Student and teacher surveys were sought in secondary 
schools only. 
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Steps to Obtain Responses in Phase 1 Principal Survey 
Initially Planned Implemented 



Initial mailing of 
900 questionnaires 



Heads up mailing to 1287 principals 
indicating questionnaire is coming 

Initial mailing of 1287 questionnaires 



return 



1213 reminder post-cards 

Telephone contact 1112 schools required calls because 
with school to seek they did not return materials without 

one; 8,783 completed phone calls; 7.9 
telephone contacts per school that 
initially failed to respond (range 1-36) 

38 requests for district approval for 
principal to answer questions filed 

964 replacement deliveries by Federal 
Express with questionnaires, personal 
note, letters from principals’ assns. 

670 ADDITIONAL replacement 
deliveries to principals who lost or 
discarded materials (531 once, 118 
twice, 21 three times) 

6 principals interviewed 

751 mailings with letter from NIJ 
director 

238 telefax requests for completion 



Replacement 
mailings of survey 
materials 



o 
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